Image processing device

ABSTRACT

An image processing device includes a modeler that creates a spherical model having a reference point of a 3D virtual space as a center, and a horizontal plane model having a distance to the center of the spherical model set based on ground distance information; and a drawing part associating coordinate values of respective vertexes on the spherical model and coordinate values of respective vertexes on the horizontal plane model with coordinate values of the omnidirectional image, replaces the coordinate values of the omnidirectional image corresponding to the coordinate values of the respective vertexes on the horizontal plane model with coordinate values of the omnidirectional image that are set to respective intersections between respective direction vectors directed from the center of the spherical model to the respective vertexes, and the spherical model, and maps the omnidirectional image on the spherical model and the horizontal plane model to form a background image Imap for the 3D virtual space.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims priority of Japanese Patent ApplicationNo. 2022-074793 filed Apr. 28, 2022. The entire contents of which arehereby incorporated by reference.

TECHNICAL FIELD

The present invention relates to an image processing device for forminga background image for a 3D virtual space.

BACKGROUND

As a technology to create a background for a virtual reality (VR) spacein a simple way, there has been known a method of pasting anomnidirectional image imaged by a spherical camera or the like on aspherical 3D model. When a background image created with this method isdisplayed on a head-mounted display (HMD), a user wearing the HMDobserves a 3D virtual space which seems as if an omnidirectional imageis projected around the user.

For example, JP 2019-133310 A discloses an image processing deviceincluding: modeling means that forms a 3D mesh-shaped model by combiningmultiple mesh shapes corresponding to the features of an omnidirectionalimage; and drawing means that converts, based on a coordinate value of avirtual reference point set in a 3D space and coordinate values ofrespective pixels of the 3D mesh-shaped model, the coordinate values ofthe respective pixels into a coordinate system of the omnidirectionalimage, and maps the omnidirectional image on the 3D mesh-shaped model toform an omnidirectional three-dimensional image. According to this imageprocessing device, it is possible to provide a virtual environment withan omnidirectional image with a three-dimensional feeling.

SUMMARY

When a background image in which an omnidirectional image is pasted onlyon a spherical 3D model is projected on the HMD, there is no informationregarding an image at a position corresponding to the ground in the 3Dvirtual space observed by the user wearing the HMD, which gives afeeling of floating to the user. In above-described Patent Literature 1,a plane mesh-shaped model is formed to correspond to a horizontal plane(the ground) in the omnidirectional image, and the omnidirectional imageis mapped on the 3D mesh-shaped model configured by combining the planemesh-shaped model and a spherical mesh-shaped model, to thereby reducethe user's feeling of floating.

However, the plane mesh-shaped model of Patent Literature 1 is formed soas to correspond to the position of the horizontal plane in theomnidirectional image regardless of the distance between the ground andthe imaging position at the time of actually imaging the omnidirectionalimage. Hence, when viewing an image of the ground mapped on the planemesh-shaped model, the user might feel something strange about thedistance to the ground in the 3D virtual space, which might decrease thesense of realism in the 3D virtual space and reduce the sense ofimmersion.

The present invention has been made in light of the above points, and ithas an object to provide an image processing device capable of creatinga background image having a high sense of immersion in a 3D virtualspace, using an omnidirectional image.

In order to attain the above object, one aspect of the present inventionprovides an image processing device including: a storage part thatstores an omnidirectional image acquired by imaging a reality spaceincluding a ground and a three-dimensional object by a spherical camera;a modeling part that forms a 3D model configured by combining multiplemeshes based on features of the omnidirectional image stored in thestorage part; and a drawing part that converts the coordinate values ofthe respective vertexes on the 3D model formed by the modeling part intoa coordinate system of the omnidirectional image stored in the storagepart based on a coordinate value of a reference point set in the 3Dvirtual space, and maps the omnidirectional image on the 3D model toform an omnidirectional three-dimensional image. In this imageprocessing device, the storage part is configured to store grounddistance information indicating a distance from an imaging position ofthe spherical camera to the ground. The modeling part is configured toform a spherical model having the reference point in the 3D virtualspace as a center, and to form a horizontal plane model that is arrangeddownward when viewed from the center of the spherical model and has adistance to the center set based on the ground distance informationstored in the storage part. The drawing part is configured to, based onthe coordinate value of the reference point in the 3D virtual space,convert coordinate values of respective vertexes on the spherical modelformed by the modeling part into a coordinate system of theomnidirectional image stored in the storage part, replace the coordinatevalues of the respective vertexes on the horizontal plane model formedby the modeling part with coordinate values of the converted coordinatesystem that are set to respective intersections between respectivedirection vectors directed from the center of the spherical model to therespective vertexes on the horizontal plane model, and the sphericalmodel, and map the omnidirectional image on the spherical model and thehorizontal plane model to form the omnidirectional three-dimensionalimage.

According to the image processing device of the present invention, sincea user's feeling of strangeness about the distance to the ground in a 3Dvirtual space can be reduced, it is possible to create a backgroundimage with a high sense of immersion in the 3D virtual space, using anomnidirectional image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing a schematic configuration of a drivingsimulator system to which an image processing device according to thefirst embodiment of the present invention is applied.

FIG. 2 is a conceptual diagram showing a 3D model (a spherical model anda horizontal plane model) formed by a modeling part in the firstembodiment.

FIG. 3 is a conceptual diagram for explaining processing, executed by adrawing part in the first embodiment, of associating coordinate valuesof respective vertexes on the spherical model with coordinate values ofan omnidirectional image.

FIG. 4 is a conceptual diagram for explaining processing, executed bythe drawing part in the first embodiment, of replacing UV coordinatevalues corresponding to coordinate values of respective vertexes on thehorizontal plane model with UV coordinate values set on the sphericalmodel.

FIG. 5 is a conceptual diagram showing a 3D model (the spherical model,the horizontal plane model, and a vertical plane model) formed by themodeling part of the image processing device according to the secondembodiment of the present invention.

FIG. 6 is a conceptual diagram for explaining processing, executed bythe drawing part in the second embodiment, of replacing UV coordinatevalues corresponding to coordinate values of respective vertexes on thevertical plane model with UV coordinate values set on the sphericalmodel.

FIG. 7 is a conceptual diagram for explaining effects caused by thesecond embodiment.

FIG. 8 is a conceptual diagram showing a cross section of the 3D modelon which the omnidirectional image is pasted, in the third embodiment.

FIG. 9 is a conceptual diagram showing a modification related to thesecond embodiment.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present invention will be described indetail with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a schematic configuration of a drivingsimulator system to which an image processing device according to thefirst embodiment of the present invention is applied.

In FIG. 1 , the driving simulator system 1 includes: a spherical camera2 that images a reality space; an image processing device 3 according tothe first embodiment that creates a background image Imap for a 3Dvirtual space using an omnidirectional image I imaged by the sphericalcamera 2; and a head-mounted display (HMD) 4 that displays thebackground image Imap created by the image processing device 3. Thisdriving simulator system 1 can be used, for example, for a simulatedexperience of a vehicle driving in vehicle developments for automobilesor the like and in sales promotion activities.

The image processing device 3 includes: an input part 31, a storage part32, a modeling part 33, a drawing part 34, and an output part 35, asfunction blocks thereof, for example. Although illustration of thehardware configuration of the image processing device 3 is omitted here,the image processing device 3 includes, for example, a computer systemincluding a processor, a memory, a user input interface, and acommunication interface. That is, in the image processing device 3, afunction of each block is realized by reading and executing a programstored in the memory by the processor of the computer system.

The input part 31 is realized by the user input interface of thecomputer system, and includes, for example, a keyboard, a mouse, anoperation controller, and others. The input part 31 also includes areceiver that receives information from the outside by wire orwirelessly, and functions as an external information input interfacethat receives information from an external computer or the like. Theinput part 31 is provided with an omnidirectional image I acquired byimaging the reality space by the spherical camera 2, and theomnidirectional image I is stored in the storage part 32. Note that thespherical camera 2 is also called an omnidirectional camera or a360-degree camera.

The reality space imaged by the spherical camera 2 includes the ground,such as a road on which a vehicle travels and a sidewalk of the road,and three-dimensional objects such as buildings, traffic signals,traffic signs, and forests located around the road. At the time ofimaging the omnidirectional image I, a distance D1 from an imagingposition of the spherical camera 2 to the ground, and a distance D2 fromthe imaging position of the spherical camera 2 to a three-dimensionalobject are measured, and information indicating a measurement result isstored in the storage part 32 together with the omnidirectional image Ivia the input part 31. That is, the storage part 32 of the imageprocessing device 3 is configured to be able to store theomnidirectional image I acquired by imaging the reality space includingthe ground and the three-dimensional object by the spherical camera 2,information indicating the distance D1 to the ground (hereinafter,referred to as “ground distance information”), and informationindicating the distance D2 to the three-dimensional object (hereinafter,referred to as “three-dimensional object distance information”), thedistances having been measured at the time of imaging theomnidirectional image I.

The modeling part 33 forms a 3D model, which is configured by combiningmultiple meshes based on the features of the omnidirectional image Istored in the storage part 32. Each of the multiple meshes configuringthe 3D model has three or more vertexes, edges connecting the vertexes,and faces closed by the edges. The density of the multiple meshes issettable in accordance with the accuracy required for the 3D model. Inthe modeling part 33 of the first embodiment, as the 3D model includingthe above multiple meshes, a spherical model Ms and a horizontal planemodel Mh corresponding to the ground are formed, for example.

FIG. 2 is a conceptual diagram showing the spherical model Ms and thehorizontal plane model Mh formed by the modeling part 33.

As shown in FIG. 2 , the spherical model Ms is 3D model data configuredby combining multiple meshes (not shown) into a spherical surface with aradius r, while a reference point set in a three-dimensional virtualspace is set as a center C. The 3D virtual space is a virtual spaceprovided to a user U who wears the HMD 4. The reference point of the 3Dvirtual space is set to correspond to the eye position of the user U inthe 3D virtual space. The omnidirectional image I is mapped on thespherical model Ms with the center C set to the reference point of the3D virtual space, and is projected on the HMD 4, to thereby allow theuser U to observe the 3D virtual space from a viewpoint corresponding tothe imaging position of the spherical camera 2. The setting of theradius r of the spherical model Ms will be described later.

The horizontal plane model Mh is 2D model data that is arranged downwardwhen viewed from the center C of the spherical model Ms (the referencepoint of the 3D virtual space), as shown in a shaded region in FIG. 2 ,and has a plane extending generally in a horizontal direction, which isconfigured by combining multiple meshes. A distance Dh from the center Cof the spherical model Ms to the horizontal plane model Mh, that is, alength of a perpendicular line extending downward from the center C ofthe spherical model Ms to the horizontal plane model Mh is set based onthe ground distance information stored in the storage part 32.

Specifically, the distance Dh from the center C of the spherical modelMs to the horizontal plane model Mh is set to correspond to the distanceD1 from the imaging position of the spherical camera 2 to the ground,which is indicated by the ground distance information. In other words,as the installation height of the spherical camera 2 from the ground atthe time of imaging the omnidirectional image I becomes higher (orlower), the distance Dh from the center C of the spherical model Ms tothe horizontal plane model Mh becomes longer (or shorter). Thehorizontal plane model Mh thus configured is combined with the sphericalmodel Ms, to thereby generate 3D model data as a whole. The 3D model(the spherical model Ms and the horizontal plane model Mh) formed by themodeling part 33 is sent to the drawing part 34.

The drawing part 34 executes, based on the coordinate value of thereference point in the 3D virtual space, processing of associatingcoordinate values of respective vertexes on the spherical model Msformed by the modeling part 33 with coordinate values of theomnidirectional image I stored in the storage part 32, and associatingcoordinate values of respective vertexes on the horizontal plane modelMh formed by the modeling part 33 with the coordinate values of theomnidirectional image I stored in the storage part 32. In addition, thedrawing part 34 executes processing of replacing the coordinate valuesof the omnidirectional image I corresponding to the coordinate values ofthe respective vertexes on the horizontal plane model Mh with coordinatevalues of the omnidirectional image I that are set to respectiveintersections between respective direction vectors directed from thecenter C of the spherical model Ms to the respective vertexes on thehorizontal plane model Mh, and the spherical model Ms. As a result, thedrawing part 34 maps the omnidirectional image I on the spherical modelMs and the horizontal plane model Mh to form the background image Imapfor the 3D virtual space, and outputs the background image Imap to theHMD 4 via the output part 35.

FIG. 3 is a conceptual diagram for explaining the processing, executedby the drawing part 34, of associating the coordinate values of therespective vertexes on the spherical model Ms with the coordinate valuesof the omnidirectional image I.

As shown in the right part in FIG. 3 , the respective vertexes (notshown) of multiple meshes configuring the spherical model Ms exist on aspherical surface with the radius r while the reference point of the 3Dvirtual space is set as the center C. In an orthogonal coordinate systemincluding three X, Y, and Z axes that are mutually orthogonal(hereinafter referred to as an “XYZ coordinate system”), when thecoordinate value of the reference point of the 3D virtual space is setto (a, b, c), the coordinate value (x, y, z) of each vertex on thespherical model Ms can be expressed by using a spherical equation:(x−a)²+(y−b)²+(z−c)²=r².

In addition, as for the coordinate system of the omnidirectional imageI, as shown in the left part in FIG. 3 , when a texture (an image) ismapped on the 3D model, a UV coordinate system used for specifying apasting position, direction, size, etc., is set. The omnidirectionalimage I in this UV coordinate system is mapped on the 3D model (thespherical model Ms and the horizontal plane model Mh) formed by themodeling part 33. That is, in the image mapped on the 3D model,coordinate values of the UV coordinate system are uniquely set withrespect to the coordinate values of the XYZ coordinate system at therespective vertexes on the 3D model. In the first embodiment, the imagemapped on the spherical model Ms has UV coordinate values set in aone-to-one relationship with respect to the XYZ coordinate values of therespective vertexes on the spherical model Ms, as described above. Inaddition, the image mapped on the horizontal plane model Mh has UVcoordinate values set in a one-to-one relationship with respect to theXYZ coordinate values of the respective vertexes on the horizontal planemodel Mh, as described later.

Specifically, in the image mapped on the spherical model Ms (theomnidirectional image I), the UV coordinate system, which is a 2Dorthogonal coordinate system having a horizontal axis of U and avertical axis of V, is set (the left part in FIG. 3 ). Vertexes in anequidistant grid pattern are set on the omnidirectional image I, and UVcoordinate values of the respective vertexes are associated respectivelywith the XYZ coordinate values of the respective vertexes on thespherical model Ms. Each UV coordinate value is set within a range of 0to 1 for each direction of the U axis and the V axis. By setting thevertexes in an equidistant grid pattern on the omnidirectional image Iin the above manner, distortion in the omnidirectional image Ito bepasted on the spherical model Ms is reduced. Note that, for convenienceof creating the meshes on the spherical model Ms, in the vicinity ofpoles serving as endpoints (areas surrounded by dashed lines in theright part in FIG. 3 ), vertexes in a triangular wavy pattern ratherthan in a grid pattern are set in corresponding regions of theomnidirectional image I. Hence, in the vicinity of the poles of thespherical model Ms, slight areas that cannot be pasted on the sphericalmodel Ms are generated in the omnidirectional image I. Hatched regionsin the left part in FIG. 3 represent regions of the omnidirectionalimage Ito be pasted on the spherical model Ms.

FIG. 4 is a conceptual diagram for explaining processing, executed bythe drawing part 34, of replacing the UV coordinate values correspondingto the coordinate values of the respective vertexes on the horizontalplane model Mh with the UV coordinate values set on the spherical modelMs. Note that FIG. 4 shows respective cross sections taken along the XZplane of the spherical model Ms and the horizontal plane model Mh inFIG. 2 .

As shown in the upper part in FIG. 4 , the horizontal plane model Mh isprovided at a position downward by the distance Dh from the center C ofthe spherical model Ms. Respective vertexes Ph (thin white circle marks)of the meshes configuring the horizontal plane model Mh are arranged onthe horizontal plane model Mh. The coordinate value (x, y, z) of the XYZcoordinate system in the 3D virtual space is set to each vertex Ph onthe horizontal plane model Mh. As described above, when the coordinatevalue of the reference point of the 3D virtual space is set to (a, b,c), the coordinate value of each vertex Ph in the Z-axis direction isexpressed by z=c−Dh.

The image thus mapped on the horizontal plane model Mh has the UVcoordinate values set in a one-to-one relationship with respect to theXYZ coordinate values of the respective vertexes on the horizontal planemodel Mh, as described above. In the first embodiment, the drawing part34 executes processing of replacing the UV coordinate valuescorresponding to the XYZ coordinate values of the respective vertexes onthe horizontal plane model Mh with the UV coordinate values that are setto respective intersections between respective direction vectorsdirected from the center C of the spherical model Ms to the respectivevertexes on the horizontal plane model Mh, and the spherical model Ms.In this processing, as shown in the middle part in FIG. 4 , first, thedrawing part 34 sets straight lines connecting the center C of thespherical model Ms and the respective vertexes Ph on the horizontalplane model Mh, and extends the straight lines in a radially outwarddirection (dotted lines in FIG. 4 ). These straight lines correspond tothe direction vectors directed from the center C of the spherical modelMs to the respective vertexes Ph on the horizontal plane model Mh.

The drawing part 34 identifies respective intersections Ps (bold whitecircle marks) between the respective straight lines (direction vectors)and the spherical model Ms. The UV coordinate values of theomnidirectional image I mapped on the spherical model Ms are set to therespective intersections Ps. Then, the drawing part 34 acquires the UVcoordinate value set to the respective intersections Ps, and replacesthe UV coordinate values corresponding to the XYZ coordinate values ofthe respective vertexes Ph on the horizontal plane model with theacquired UV coordinate values, as shown in the lower part in FIG. 4 . Asfor UV coordinate values corresponding to the XYZ coordinate values ofsome vertexes Ph among the respective vertexes Ph on the horizontalplane model Mh, these some vertexes Ph arranged in a region in which thespherical model Ms overlaps with the horizontal plane model Mh, thoseconcerned UV coordinate values are replaced with UV coordinate valuesset to points on the spherical model Ms that coincide with these somevertexes Ph. Through the above series of processings, an imagecorresponding to the ground in the omnidirectional image I is pasted onthe horizontal plane model Mh.

The background image Imap for the 3D virtual space thus formed bymapping the omnidirectional image I on the spherical model Ms and thehorizontal plane model Mh by the drawing part 34 is sent to the outputpart 35 (FIG. 1 ) of the image processing device 3. The output part 35outputs the background image Imap from the drawing part 34 to the HMD 4using the communication interface of the computer system.

The HMD 4 is a well-known display device configured to project thebackground image Imap from the image processing device 3 as thebackground for the 3D virtual space so as to allow the user U toperceive a three-dimensional virtual space. As for the HMD 4, a headtracking function to detect a movement of the head of the user U and aposition tracking function to detect a movement of the body of the userU may be realized by using sensors built in the HMD 4, or the like.Information on the movement of the user U detected by the abovefunctions is transmitted to the drawing part 34 via the input part 31 ofthe image processing device 3, and is used for updating the backgroundimage Imap linked to the movement of the user U.

In the driving simulator system 1 to which the above-described imageprocessing device 3 according to the first embodiment is applied, thebackground image Imap for the 3D virtual space is formed by mapping theomnidirectional image I that is acquired by imaging the reality space bythe spherical camera 2 on the spherical model Ms and the horizontalplane model Mh by the image processing device 3, and the backgroundimage Imap is projected on the HMD 4. Accordingly, the user U wearingthe HMD 4 can observe the background image Imap for the 3D virtual spaceprojected on the HMD 4 from a viewpoint corresponding to the imagingposition of the spherical camera 2.

At this time, since the distance Dh from the horizontal plane model Mhcorresponding to the ground to the center C of the spherical model Ms(the reference point of the 3D virtual space) is set to correspond tothe distance D1 from the imaging position of the spherical camera 2 tothe ground at the time of imaging the omnidirectional image I, the userU who views the image of the ground mapped on the horizontal plane modelMh can have less of a feeling of strangeness about the distance to theground in the 3D virtual space. Therefore, according to the imageprocessing device 3 of the first embodiment, it is possible to createthe background image with a high sense of immersion in the 3D virtualspace, using the omnidirectional image I imaged by the spherical camera2. In particular, by associating the XYZ coordinate values of therespective vertexes on the spherical model Ms and the horizontal planemodel Mh with the UV coordinate values of the omnidirectional image I,it is possible to realize highly versatile mapping processing.

Next, the image processing device according to the second embodiment ofthe present invention will be described.

In the above-described image processing device 3 according to the firstembodiment, there has been described the case in which the modeling part33 forms the 3D model by combining the spherical model Ms and thehorizontal plane model Mh corresponding to the ground. In the imageprocessing device 3 according to the second embodiment, there will bedescribed the case in which the modeling part 33 forms a vertical planemodel Mv corresponding to a three-dimensional object, in addition to thespherical model Ms and the horizontal plane model Mh.

FIG. 5 is a conceptual diagram showing a 3D model (the spherical modelMs, the horizontal plane model Mh, and the vertical plane model Mv)formed by the modeling part 33 in the second embodiment. Note that thefunction blocks of the image processing device 3 according to the secondembodiment, and the configuration of the driving simulator system 1 towhich the image processing device 3 according to the second embodimentis applied are the same as the configuration of the first embodimentshown in FIG. 1 described above and therefore, description thereof isomitted here.

As shown in FIG. 5 , the modeling part 33 in the second embodiment formsthe spherical model Ms and the horizontal plane model Mh in the samemanner as in the first embodiment described above, and also forms thevertical plane model Mv corresponding to the three-dimensional objectincluded in the omnidirectional image I. The vertical plane model Mv is,as shown in a shaded region in FIG. 5 , 2D model data that is arrangedin a lateral direction when viewed from the center C of the sphericalmodel Ms (the reference point of the 3D virtual space) and has a planeextending generally in a vertical direction and configured by combiningmultiple meshes. In an example shown in FIG. 5 , the vertical planemodel Mv is spaced in the X-axis direction with respect to the center Cof the spherical model Ms and is formed substantially parallel to the YZplane. A distance Dv from the center C of the spherical model Ms to thevertical plane model Mv is set based on the three-dimensional objectdistance information stored in the storage part 32.

Specifically, the distance Dv from the center C of the spherical modelMs to the vertical plane model My is set to correspond to the distanceD2 from the imaging position of the spherical camera 2 to thethree-dimensional object, which is indicated by the three-dimensionalobject distance information. In other words, as the three-dimensionalobject becomes farther (or closer) from the installation position of thespherical camera 2 at the time of imaging the omnidirectional image I,the distance Dv from the center C of the spherical model Ms to thevertical plane model My becomes longer (or shorter). The vertical planemodel My thus configured is combined with the spherical model Ms and thehorizontal plane model Mh, to thereby create 3D model data as a whole.The 3D model (the spherical model Ms, the horizontal plane model Mh, andthe vertical plane model Mv) formed by the modeling part 33 is sent tothe drawing part 34.

In the same manner as in the above first embodiment, the drawing part 34in the second embodiment executes processing of replacing the UVcoordinate values corresponding to the XYZ coordinate values of therespective vertexes on the horizontal plane model Mh with the UVcoordinate value set to respective intersections between respectivedirection vectors directed from the center C of the spherical model Msto the respective vertexes on the horizontal plane model Mh, and thespherical model Ms, and also executes processing of replacing the UVcoordinate values corresponding to the XYZ coordinate values ofrespective vertexes on vertical plane model My with the UV coordinatevalues set to respective intersections between respective directionvectors directed from the center C of the spherical model Ms to therespective vertexes on the vertical plane model Mv, and the sphericalmodel Ms. Then, the drawing part 34 maps the omnidirectional image I onthe spherical model Ms, the horizontal plane model Mh, and the verticalplane model My to form the background image Imap for the 3D virtualspace, and outputs the background image Imap to the HMD 4 via the outputpart 35.

FIG. 6 is a conceptual diagram for explaining processing, executed bythe drawing part 34 in the second embodiment, of replacing the UVcoordinate values corresponding to the coordinate values of therespective vertexes on the vertical plane model Mv with the UVcoordinate values set on the spherical model Ms. Note that FIG. 6 showsa cross section taken along the XZ plane of the spherical model Ms, thehorizontal plane model Mh, and the vertical plane model Mv in FIG. 5 .

As shown in the upper part in FIG. 6 , the vertical plane model Mv isprovided at a position apart by a distance Dv from the center C of thespherical model Ms in the lateral direction (the x-axis direction).Respective vertexes Pv (thin white circle marks) of the meshesconfiguring the vertical plane model Mv are arranged on the verticalplane model Mv. The coordinate value (x, y, z) of the XYZ coordinatesystem in the 3D virtual space is set to each vertex Pv on the verticalplane model Mv. As with the above first embodiment, when the coordinatevalue of the reference point of the 3D virtual space is set to (a, b,c), the coordinate value of each vertex Pv in the x-axis direction isexpressed by x=a−Dv.

The image thus mapped on the vertical plane model Mv has the UVcoordinate values set in a one-to-one relationship with respect to theXYZ coordinate values of the respective vertexes on the vertical planemodel Mv. In the second embodiment, the drawing part 34 executesprocessing of replacing the UV coordinate values corresponding to theXYZ coordinate values of the respective vertexes on the vertical planemodel Mv with the UV coordinate values set to respective intersectionsbetween respective direction vectors directed from the center C of thespherical model Ms to the respective vertexes on the vertical planemodel Mv, and the spherical model Ms. In this processing, as shown inthe middle part in FIG. 6 , first, the drawing part 34 sets straightlines connecting the center C of the spherical model Ms respectively tothe respective vertexes Pv on the vertical plane model Mv, and extendsthe straight lines in a radially outward direction (dotted lines in FIG.6 ). These straight lines correspond to the direction vectors directedfrom the center C of the spherical model Ms to the respective vertexesPv on the vertical plane model Mv.

The drawing part 34 identifies respective intersections Ps (bold whitecircle marks) between the respective straight lines (the directionvectors) and the spherical model Ms. The UV coordinate values of theomnidirectional image I mapped on the spherical model Ms are set to therespective intersections Ps. Therefore, the drawing part 34 acquires theUV coordinate value set to the respective intersections Ps, and replacesthe UV coordinate values corresponding to the XYZ coordinate values ofthe respective vertexes Pv on the vertical plane model with the acquiredUV coordinate values, as shown in the lower part in FIG. 6 . As for UVcoordinate values corresponding to the XYZ coordinate values of somevertexes Pv among the respective vertexes Pv on the vertical plane modelMv, these some vertexes Pv being arranged outward of the spherical modelMs, those concerning UV coordinate values are replaced with UVcoordinate values set to intersections Ps on the spherical model Ms thatare located on line segments connecting these some vertexes Pv and thecenter C of the spherical model Ms. Through the above series ofprocessings, an image region corresponding to the three-dimensionalobject in the omnidirectional image I is pasted on the vertical planemodel Mv.

In the driving simulator system 1 to which the above-described imageprocessing device 3 according to the second embodiment is applied, thebackground image Imap for the 3D virtual space is formed by mapping theomnidirectional image I that is an image of the reality space imaged bythe spherical camera 2 on the spherical model Ms, the horizontal planemodel Mh, and the vertical plane model Mv by the image processing device3 to form the background image Imap for the 3D virtual space, and thebackground image Imap is projected on the HMD 4. Since the verticalplane model Mv corresponding to the three-dimensional object is formedand the omnidirectional image I is mapped thereon, it is possible tocreate a background image for the 3D virtual space with less sense ofdiscomfort. FIG. 7 is a conceptual diagram for explaining effects causedby the second embodiment.

As shown in the upper part and the middle part in FIG. 7 , when thebackground image for the 3D virtual space is created using theomnidirectional image I acquired by imaging the reality space includingthe ground G and the three-dimensional object O by the spherical camera2, in the case of the first embodiment in which only the spherical modelMs and the horizontal plane model Mh are formed, as the position of thethree-dimensional object O becomes farther from the imaging position ofthe spherical camera 2, coarseness of the meshes of the 3D model onwhich an image Io of the three-dimensional object is pasted becomes moreapparent, and the image Io of the three-dimensional object pasted on thehorizontal plane model Mh may be distorted in accordance with thedistance D2 from the three-dimensional object O to the imaging position.Such coarseness of the meshes and distortion of the image Io of thethree-dimensional object can be reduced by preparing the vertical planemodel Mv corresponding to the three-dimensional object, as shown in thelower part in FIG. 7 . In particular, by setting the distance Dv fromthe vertical plane model Mv to the center C of the spherical model Ms tocorrespond to the distance D2 from the imaging position of the sphericalcamera 2 to the three-dimensional object O, it becomes possible to pastethe image Io of the three-dimensional object on the vertical plane modelMv without distortion. Therefore, according to the image processingdevice 3 of the second embodiment, it is possible to create thebackground image with a higher sense of immersion in the 3D virtualspace, using the omnidirectional image I imaged by the spherical camera2.

Next, the image processing device according to the third embodiment ofthe present invention will be described.

In the above-described image processing device 3 according to the secondembodiment, there has been described the case in which coarseness of themeshes and distortion of the image Io of the three-dimensional objectare reduced by forming the vertical plane model Mv corresponding to thethree-dimensional object. In the image processing device 3 according tothe third embodiment, there will be described an example of adjustingthe radius r of the spherical model Ms, instead of forming the verticalplane model Mv, to thereby realize the same effects as those in thesecond embodiment. Note that the function blocks of the image processingdevice 3 according to the third embodiment and the configuration of thedriving simulator system 1 to which the image processing device 3according to the third embodiment is applied are the same as theconfiguration of the first embodiment shown in FIG. 1 described aboveand therefore, description thereof is omitted here.

FIG. 8 is a conceptual diagram showing a cross section of the 3D model(the spherical model and the horizontal plane model) taken along the X-Zplane, the 3D model on which the omnidirectional image I is pasted inthe third embodiment.

As shown in FIG. 8 , in the image processing device 3 according to thethird embodiment, in the case in which at least part of the image Iocorresponding to the three-dimensional object in the background imageImap, formed by the drawing part 34 in the same manner as in the firstembodiment described above, is mapped on the horizontal plane model Mh(the upper part in FIG. 8 ), the modeling part 33 adjusts the radius rof the spherical model Ms to r′ based on the three-dimensional objectdistance information stored in the storage part 32 (in the lower part inFIG. 8 ). Through such an adjustment of the radius of the sphericalmodel Ms to r′, the image Io corresponding to the three-dimensionalobject in the omnidirectional image I is mapped only on the sphericalmodel.

For example, assuming the case of creating a background for a 3D virtualspace based on the driver's viewpoint of a vehicle traveling on ageneral urban road, the above-described adjustment of the radius of thespherical model Ms will be specifically described as follows: a minimumroad width for a vehicle with a width of 2.5 m to travel is set to 6.5m. If a driving orientation is located at a position about 1 m outwardfrom the center in the width direction of this road, a distance from theviewpoint of the driver to an edge of the road is (6.5/2)−1.0=2.25 m.Furthermore, assuming that there is a sidewalk with a width of 1.0 moutside the road edge and that there is a three-dimensional object, suchas a house, a building, and woods, ahead of the sidewalk, the distancefrom the viewpoint of the driver to the three-dimensional object is2.25+1.0=3.25 m.

Hence, in the case of assuming that a vehicle traveling on a generalurban road, it is preferable to set the adjusted radius r′ of thespherical model Ms formed by the modeling part 33 to 3.25 m. Byadjusting the radius of the spherical model Ms based on the width of theroad on which the vehicle travels, in a typical driving scene on ageneral urban road, it is possible to project on the HMD 4 thebackground image Imap for the 3D virtual space that does not cause thedriver to feel strangeness about the appearance of the surroundingthree-dimensional object.

In addition, in the case in which the vehicle travels in an open space,such as a parking lot, it is preferable to set a distance to athree-dimensional object located closest to the spherical camera 2 tothe radius r′ of the spherical model Ms. That is, when a plurality ofthree-dimensional objects are included in the reality space imaged bythe spherical camera 2, the distance from the imaging position of thespherical camera 2 to the three-dimensional object located closest tothe spherical camera 2 among the plurality of three-dimensional objectsis measured, and three-dimensional object minimum distance informationindicating the distance is stored in the storage part 32 of the imageprocessing device 3. When the radius of the spherical model Ms isadjusted by the modeling part 33, the radius r′ of the spherical modelMs is set based on the three-dimensional object minimum distanceinformation stored in the storage part 32.

By adjusting the radius of the spherical model Ms in this manner, in atypical driving scene on a general urban road, when the vehicle travelsin an open space such as a parking lot, it is possible to project on theHMD 4 the background image Imap for the 3D virtual space that does notcause the driver to feel strangeness about the appearance of thesurrounding three-dimensional object. It should be noted that if theadjusted radius r′ of the spherical model Ms is too small, thehorizontal plane model Mh corresponding to the ground is located outsidethe spherical model Ms, so that the 3D model used for creating thebackground for the 3D virtual space may become inappropriate.

Although the first to third embodiments of the present invention havebeen described above, the present invention is not limited to theabove-described embodiments, and various modifications and changes canbe made on the basis of the technical idea of the present invention. Forexample, in the first to third embodiments described above, althoughthere has been exemplified an example of setting the UV coordinatesystem in the omnidirectional image I, a coordinate system other thanthe UV coordinate system may be set in the omnidirectional image I.

In the second embodiment described above, although there has beendescribed the case in which only one vertical plane model Mvcorresponding to the three-dimensional object is formed, two or morevertical plane models Mv may be formed. As a specific example, in thecase in which a three-dimensional object is placed only in theleft-right direction of the road on which the vehicle travels in atunnel, an urban area, or the like, as shown in FIG. 9 , it may beconfigured such that, after the radius r of the spherical model Ms isset to an appropriate size, a vertical plane model Mv′ similar to thevertical plane model Mv in the second embodiment described above isformed on the side opposite to the vertical plane model My across thecenter C of the spherical model Ms, and the omnidirectional image I ismapped on the left and right vertical plane models Mv, Mv′.

1. An image processing device, comprising: a storage part that stores anomnidirectional image acquired by imaging a reality space including aground and a three-dimensional object by a spherical camera; a modelingpart that forms a 3D model configured by combining multiple meshes basedon features of the omnidirectional image stored in the storage part; anda drawing part that maps the omnidirectional image stored in the storagepart on the 3D model formed by the modeling part based on a coordinatevalue of a reference point set in a 3D virtual space so as to form abackground image for the 3D virtual space, characterized in that thestorage part is configured to store ground distance informationindicating a distance from an imaging position of the spherical camerato the ground, the modeling part is configured to form a spherical modelhaving the reference point in the 3D virtual space as a center, and toform a horizontal plane model that is arranged downward when viewed fromthe center of the spherical model and has a distance to the center setbased on the ground distance information stored in the storage part, andthe drawing part is configured to, based on the coordinate value of thereference point in the 3D virtual space, associate coordinate values ofrespective vertexes on the spherical model formed by the modeling partwith coordinate values of the omnidirectional image stored in thestorage part, associate coordinate values of respective vertexes on thehorizontal plane model formed by the modeling part with the coordinatevalues of the omnidirectional image stored in the storage part, andreplace the coordinate values of the omnidirectional image correspondingto the coordinate values of the respective vertexes on the horizontalplane model with coordinate values of the omnidirectional image that areset to respective intersections between respective direction vectorsdirected from the center of the spherical model to the respectivevertexes on the horizontal plane model, and the spherical model so as tomap the omnidirectional image on the spherical model and the horizontalplane model to form the background image for the 3D virtual space. 2.The image processing device according to claim 1, wherein a UVcoordinate value in a UV coordinate system is set to each pixel of theomnidirectional image, and the drawing part is configured to set the UVcoordinate values of the omnidirectional image corresponding to thecoordinate values of the respective vertexes depending on a positionalrelationship of the reference point in the 3D virtual space with respectto the respective vertexes on the spherical model and the respectivevertexes on the horizontal plane model, to extend straight linesconnecting the center of the spherical model respectively to therespective vertexes on the horizontal plane model in a radially outwarddirection, to acquire UV coordinate values set to respectiveintersections between the respective straight lines and the sphericalmodel, and to replace the UV coordinate values of the omnidirectionalimage corresponding to the coordinate values of the respective vertexeson the horizontal plane model with the acquired UV coordinate values. 3.The image processing device according to claim 1, wherein the storagepart is configured to store three-dimensional object distanceinformation indicating a distance from the imaging position of thespherical camera to the three-dimensional object, the modeling part isconfigured to form a vertical plane model that is arranged in a lateraldirection when viewed from the center of the spherical model and is setbased on the three-dimensional object distance information including thedistance to the center stored in the storage part, and the drawing partis configured to, based on the coordinate value of the reference pointin the 3D virtual space, associate coordinate values of respectivevertexes on the vertical plane model formed by the modeling part withthe coordinate values of the omnidirectional image stored in the storagepart, replace the coordinate values of the omnidirectional imagecorresponding to the coordinate values of the respective vertexes on thevertical plane model with coordinate values of the omnidirectional imagethat are set to respective intersections between respective directionvectors directed from the center of the spherical model to therespective vertexes on the vertical plane model, and the spherical modelso as to map the omnidirectional image on the spherical model, thehorizontal plane model, and the vertical plane model to form thebackground image for the 3D virtual space.
 4. The image processingdevice according to claim 3, wherein a UV coordinate value in a UVcoordinate system is set to each pixel of the omnidirectional image, andthe drawing part is configured to, depending on a positionalrelationship of the reference point in the 3D virtual space with respectto respective vertexes on the spherical model, respective vertexes onthe horizontal plane model, and respective vertexes on the verticalplane model, set UV coordinate values of the omnidirectional imagecorresponding to the coordinate values of the respective vertexes,extend straight lines connecting the center of the spherical modelrespectively to the respective vertexes on the horizontal plane modeland the respective vertexes on the vertical plane model in a radiallyoutward direction, acquire UV coordinate values set to respectiveintersections between the respective straight lines and the sphericalmodel, and replace the UV coordinate values of the omnidirectional imagecorresponding to the coordinate values of the respective vertexes on thehorizontal plane model and the coordinate values of the respectivevertexes on the vertical plane model with the acquired UV coordinatevalues.
 5. The image processing device according to claim 3, whereinwhen at least part of an image corresponding to the three-dimensionalobject in the background image formed by the drawing part is mapped onthe horizontal plane model, the modeling part is configured to adjust aradius of the spherical model based on the three-dimensional objectdistance information stored in the storage part so as to entirely mapthe image corresponding to the three-dimensional object on the sphericalmodel.
 6. The image processing device according to claim 5, wherein themodeling part is configured to adjust the radius of the spherical modelbased on a width of a road on which a vehicle travels.
 7. The imageprocessing device according to claim 5, wherein when images of multiplethree-dimensional objects are included in the omnidirectional image, thestorage part is configured to store three-dimensional object minimumdistance information indicating a distance from the imaging position ofthe spherical camera to an object located closest to the sphericalcamera among the multiple three-dimensional objects, and the modelingpart is configured to adjust the radius of the spherical model based onthe three-dimensional object minimum distance information that is storedin the storage part.